Frame rate converting apparatus, pan/tilt determining apparatus, and video apparatus

ABSTRACT

A frame rate converting apparatus includes a pan/tilt determining unit which determines whether an image is in a pan/tilt state on the basis of information related to motion of the image detected by a motion detecting unit for each frame of a moving image, a moving distance setting unit which sets 0 as a moving distance of an image with respect to a frame the image of which is not determined as an image in a pan/tilt state by the pan/tilt determining unit, which calculates a moving distance of an image from the information related to the motion of the image detected by the motion detecting unit with respect to a frame the image of which is determined as an image in a pan/tilt state, and which sets the obtained moving distance as a moving distance of the image, and a prediction image generating unit which generates a prediction image necessary for frame rate conversion on the basis of the moving distance of the image set for each frame by the moving distance setting unit.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a frame rate converting apparatus toconvert a frame rate of a moving image, a pan/tilt determining apparatuswhich determines whether an image is in a pan/tilt state, a videoapparatus having a frame rate converting apparatus, and a videoapparatus having a pan/tilt determining apparatus. In this case, thevideo apparatus includes a digital camera, a video camera, a televisionreceiver, and the like.

2. Description of the Related Art

For example, a moving image is picked up by a digital camera, a framerate of a recording image recorded on a recording medium is 30frames/second, and a frame rate of a reproducing image is 60frames/second. For this reason, when the reproducing image is generatedfrom the recording image, a frame rate converting process.

As a conventional frame rate converting method, as shown in FIG. 15, amethod of inverting an early frame of adjacent frames of a recordingimage is inserted between the adjacent frames. In addition, a method ofinserting a predicted screen between the adjacent frames of therecording image is developed (see Japanese Unexamined Patent PublicationNo. 9-200770 and Japanese Unexamined Patent Publication No. 2003-69961).However, in a conventional art, since the frame rate is convertedwithout considering whether the image is in a pan/tilt state, it isdisadvantageous that a smooth reproducing image cannot be obtained.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a frame rateconverting apparatus which can obtain a smooth reproducing image.

It is another object of the present invention to provide a videoapparatus having a frame rate converting apparatus which can obtain asmooth reproducing image.

It is still another object of the present invention to provide apan/tilt determining apparatus which can determine whether an image isin a pan/tilt state.

It is still another object of the present invention to provide a videoapparatus having a pan/tilt determining apparatus.

It is still another object of the present invention to provide a framerate converting apparatus which can generate a prediction image withoutgenerating a blank portion.

It is still another object of the present invention to provide a videoapparatus having a frame rate converting apparatus which can generate aprediction image without generating a blank portion.

According to the present invention, there is provided a first frame rateconverting apparatus which converts a frame rate of a moving image,including: motion detecting means which detects information related tomotion of an image for each frame of a moving image to be converted;pan/tilt determining means which determines whether the image is in apan/tilt state on the basis of the information related to the motion ofthe image detected by the motion detecting means for each frame of themoving image to be converted; moving distance setting means which sets 0as a moving distance of an image with respect to a frame the image ofwhich is not determined as an image in a pan/tilt state by the pan/tiltdetermining means, which calculates a moving distance of an image fromthe information related to the motion of the image detected by themotion detecting means with respect to a frame the image of which isdetermined as an image in a pan/tilt state, and which sets the obtainedmoving distance as a moving distance of the image; and prediction imagegenerating means which generates a prediction image necessary for framerate conversion on the basis of the moving distance of the image set foreach frame by the moving distance setting means.

According to the present invention, there is provided a second framerate converting apparatus which converts a frame rate of a moving image,including: motion detecting means which detects information related tomotion of an image for each frame of a moving image to be converted;pan/tilt determining means which determines whether the image is in apan/tilt state on the basis of the information related to the motion ofthe image detected by the motion detecting means for each frame of themoving image to be converted; moving distance setting means which sets 0as a moving distance of an image with respect to a frame the image ofwhich is not determined as an image in a pan/tilt state by the pan/tiltdetermining means, which calculates a moving distance of an image fromthe information related to the motion of the image detected by themotion detecting means with respect to a frame the image of which isdetermined as an image in a pan/tilt state, and which sets the obtainedmoving distance as a moving distance of the image; time-directionsmoothing means which smoothes the moving distance of the image set foreach frame by the moving distance setting means in a direction of time;and prediction image generating means which generates a prediction imagenecessary for frame rate conversion on the basis of the moving distanceof the image obtained by the time-direction smoothing means.

In the first or second frame rate converting apparatus, used as themotion detecting means is, for example, means which calculates a motionvector, a minimum value of correlative accumulated values, and anaverage value of the correlative accumulated values as informationrelated to the motion of the image for each of a plurality of motionvector detecting regions set in a video area of each frame by a typicalpoint matching method, and, used as the pan/tilt determining means is,for example, means having means which specifies a region in which thedetected motion vector has high reliability of the motion vectordetecting regions on the basis of the average value of the correlativeaccumulated values detected for each of the motion vector detectingregions and which calculates the number of detection regions in whichthe detected motion vectors have high reliability, means whichdetermines whether the number of detection regions in which the detectedmotion vectors have high reliability is not less than a first thresholdvalue, means which, when the number of detection regions in which thedetected moving vectors have high reliability is less than the firstthreshold value, determines that the frame is not panned/tilted, meanswhich, when the number of detected regions in which the detected motionvectors have high reliability is not less than the first thresholdvalue, specifies a detection region in which motions of a plurality oftypes are present and a detection region in which motions of a pluralityof types are not present in the detection regions in which the motionvectors have high reliability on the basis of an average value ofcorrelative accumulated values detected for the detection regions inwhich the motion vectors have high reliability and a minimum value ofthe correlative accumulated values, and which calculates the number ofdetection regions in which the motions of the plurality of types arepresent, means which determines whether the number of detection regionsin which the motions of the plurality of types are present is less thana second threshold value, means which, when the number of detectionregions in which the motions of the plurality of types are present isnot less than the second threshold value, determines that the frame isnot panned/tilted, and means which, when the number of detection regionsin which the motions of the plurality of types are present is less thanthe second threshold value, determines that the frame is panned/tilted.

In the first or second frame rate converting apparatus, used as themotion detecting means is, for example, means which calculates a motionvector, a minimum value of correlative accumulated values, and anaverage value of the correlative accumulated values as informationrelated to the motion of the image for each of a plurality of motionvector detecting regions set in a video area of each frame by a typicalpoint matching method, and, used as the pan/tilt determining means is,for example, means having means which specifies a region in which thedetected motion vector has high reliability of the motion vectordetecting regions on the basis of the average value of the correlativeaccumulated values detected for each of the motion vector detectingregions and which calculates the number of detection regions in whichthe detected motion vectors have high reliability, means whichdetermines whether the number of detection regions in which the detectedmotion vectors have high reliability is not less than a first thresholdvalue, means which, when the number of detection regions in which thedetected moving vectors have high reliability is less than the firstthreshold value, determines that the frame is not panned/tilted, meanswhich, when the number of detection regions in which the detected motionvectors have high reliability is not less than the first thresholdvalue, specifies a detection region in which motions of a plurality oftypes are present and a detection region in which motions of a pluralityof types are not present in the detection regions in which the motionvectors have high reliability on the basis of an average value ofcorrelative accumulated values detected for the detection regions inwhich the motion vectors have high reliability and a minimum value ofthe correlative accumulated values, and which calculates the number ofdetection regions in which the motions of the plurality of types arepresent, means which determines whether the number of detection regionsin which the motions of the plurality of types are present is less thana second threshold value, means which, when the number of detectionregions in which the motions of the plurality of types are present isnot less than the second threshold value, determines that the frame isnot panned/tilted, means which, when the number of detection regions inwhich the motions of the plurality of types are present is less than thesecond threshold value, specifies detection regions in which motionvectors have similarity in the detection regions in which the motionvectors have high reliability and the motions of the plurality of typesare not present on the basis of a motion vector detected for thedetection region in which the motion vector has high reliability and themotions of the plurality of types are not present, and which calculatesthe number of detection regions in which motion vectors have similarity,means which determines whether the number of detection regions in whichmotion vectors have similarity is not less than a third threshold value,means which, when the number of detection regions which are determinedas detection regions in which the motion vectors have similarity is lessthan the third threshold value, determines that the frame is notpanned/tilted, and means which, when the number of detection regionswhich are determined as detection regions in which the motion vectorshave similarity is not less than the third threshold value, determinesthat the frame is panned/tilted.

According to the present invention, there is provided a first videoapparatus having a frame rate converting apparatus which converts aframe rate of a moving image, wherein the frame rate convertingapparatus includes: motion detecting means which detects informationrelated to motion of an image for each frame of a moving image to beconverted; pan/tilt determining means which determines whether the imageis in a pan/tilt state on the basis of the information related to themotion of the image detected by the motion detecting means for eachframe of the moving image to be converted; moving distance setting meanswhich sets 0 as a moving distance of an image with respect to a framethe image of which is not determined as an image in a pan/tilt state bythe pan/tilt determining means, which calculates a moving distance of animage from the information related to the motion of the image detectedby the motion detecting means with respect to a frame the image of whichis determined as an image in a pan/tilt state, and which sets theobtained moving distance as a moving distance of the image; andprediction image generating means which generates a prediction imagenecessary for frame rate conversion on the basis of the moving distanceof the image set for each frame by the moving distance setting means.

According to the present invention, there is provided a second videoapparatus having a frame rate converting apparatus which converts aframe rate of a moving image, wherein the frame rate convertingapparatus includes: motion detecting means which detects informationrelated to motion of an image for each frame of a moving image to beconverted; pan/tilt determining means which determines whether the imageis in a pan/tilt state on the basis of the information related to themotion of the image detected by the motion detecting means for eachframe of the moving image to be converted; moving distance setting meanswhich sets 0 as a moving distance of an image with respect to a framethe image of which is not determined as an image in a pan/tilt state bythe pan/tilt determining means, which calculates a moving distance of animage from the information related to the motion of the image detectedby the motion detecting means with respect to a frame the image of whichis determined as an image in a pan/tilt state, and which sets theobtained moving distance as a moving distance of the image;time-direction smoothing means which smoothes the moving distance of theimage set for each frame by the moving distance setting means in adirection of time; and prediction image generating means which generatesa prediction image necessary for frame rate conversion on the basis ofthe moving distance of the image obtained by the time-directionsmoothing means.

According to the present invention, there is provided a first pan/tiltdetermining apparatus which determines whether an image is in a pan/tiltstate for each frame of a moving image, including: means whichcalculates a motion vector, a minimum value of correlative accumulatedvalues, and an average value of the correlative accumulated values foreach of a plurality of motion vector detecting regions set in a videoarea of each frame of a moving image by a typical point matching method;means which specifies a region in which the detected motion vector hashigh reliability of the motion vector detecting regions on the basis ofthe average value of the correlative accumulated values detected foreach of the motion vector detecting regions and which calculates thenumber of detection regions in which the detected motion vectors havehigh reliability; means which determines whether the number of detectionregions in which the detected motion vectors have high reliability isnot less than a first threshold value; means which, when the number ofdetection regions in which the detected moving vectors have highreliability is less than the first threshold value, determines that theframe is not panned/tilted; means which, when the number of detectionregions in which the detected motion vectors have high reliability isnot less than the first threshold value, specifies a detection region inwhich motions of a plurality of types are present and a detection regionin which motions of a plurality of types are not present in thedetection regions in which the motion vectors have high reliability onthe basis of an average value of correlative accumulated values detectedfor the detection regions in which the motion vectors have highreliability and a minimum value of the correlative accumulated values,and which calculates the number of detection regions in which themotions of the plurality of types are present; means which determineswhether the number of detection regions in which the motions of theplurality of types are present is less than a second threshold value;means which, when the number of detection regions in which the motionsof the plurality of types are present is not less than the secondthreshold value, determines that the frame is not panned/tilted; andmeans which, when the number of detection regions in which the motionsof the plurality of types are present is less than the second thresholdvalue, determines that the frame is panned/tilted.

According to the present invention, there is provided a second pan/tiltdetermining apparatus which determines whether an image is in a pan/tiltstate for each frame of a moving image, including: means whichcalculates a motion vector, a minimum value of correlative accumulatedvalues, and an average value of the correlative accumulated values foreach of a plurality of motion vector detecting regions set in a videoarea of each frame of a moving image by a typical point matching method;means which specifies a region in which the detected motion vector hashigh reliability of the motion vector detecting regions on the basis ofthe average value of the correlative accumulated values detected foreach of the motion vector detecting regions and which calculates thenumber of detection regions in which the detected motion vectors havehigh reliability; means which determines whether the number of detectionregions in which the detected motion vectors have high reliability isnot less than a first threshold value; means which, when the number ofdetection regions in which the detected moving vectors have highreliability is less than the first threshold value, determines that theframe is not panned/tilted; means which, when the number of detectionregions in which the detected motion vectors have high reliability isnot less than the first threshold value, specifies a detection region inwhich motions of a plurality of types are present and a detection regionin which motions of a plurality of types are not present in thedetection regions in which the motion vectors have high reliability onthe basis of an average value of correlative accumulated values detectedfor the detection regions in which the motion vectors have highreliability and a minimum value of the correlative accumulated values,and which calculates the number of detection regions in which themotions of the plurality of types are present; means which determineswhether the number of detection regions in which the motions of theplurality of types are present is less than a second threshold value;means which, when the number of detection regions in which the motionsof the plurality of types are present is not less than the secondthreshold value, determines that the frame is not panned/tilted; meanswhich, when the number of detection regions in which the motions of theplurality of types are present is less than the second threshold value,specifies detection regions in which motion vectors have similarity inthe detection regions in which the motion vectors have high reliabilityand the motions of the plurality of types are not present on the basisof a motion vector detected for the detection region in which the motionvector has high reliability and the motions of the plurality of typesare not present, and which calculates the number of detection regions inwhich motion vectors have similarity; means which determines whether thenumber of detection regions in which motion vectors have similarity isnot less than a third threshold value; means which, when the number ofdetection regions which are determined as detection regions in which themotion vectors have similarity is less than the third threshold value,determines that the frame is not panned/tilted; and means which, whenthe number of detection regions which are determined as detectionregions in which the motion vectors have similarity is not less than thethird threshold value, determines that the frame is panned/tilted.

According to the present invention, there is provided a third videoapparatus having a pan/tilt determining apparatus which determineswhether an image is in a pan/tilt state for each frame of a movingimage, wherein the pan/tilt determining apparatus includes: means whichcalculates a motion vector, a minimum value of correlative accumulatedvalues, and an average value of the correlative accumulated values foreach of a plurality of motion vector detecting regions set in a videoarea of each frame of a moving image by a typical point matching method;means which specifies a region in which the detected motion vector hashigh reliability of the motion vector detecting regions on the basis ofthe average value of the correlative accumulated values detected foreach of the motion vector detecting regions and which calculates thenumber of detection regions in which the detected motion vectors havehigh reliability; means which determines whether the number of detectionregions in which the detected motion vectors have high reliability isnot less than a first threshold value; means which, when the number ofdetection regions in which the detected moving vectors have highreliability is less than the first threshold value, determines that theframe is not panned/tilted; means which, when the number of detectionregions in which the detected motion vectors have high reliability isnot less than the first threshold value, specifies a detection region inwhich motions of a plurality of types are present and a detection regionin which motions of a plurality of types are not present in thedetection regions in which the motion vectors have high reliability onthe basis of an average value of correlative accumulated values detectedfor the detection regions in which the motion vectors have highreliability and a minimum value of the correlative accumulated values,and which calculates the number of detection regions in which themotions of the plurality of types are present; means which determineswhether the number of detection regions in which the motions of theplurality of types are present is less than a second threshold value;means which, when the number of detection regions in which the motionsof the plurality of types are present is not less than the secondthreshold value, determines that the frame is not panned/tilted; andmeans which, when the number of detection regions in which the motionsof the plurality of types are present is less than the second thresholdvalue, determines that the frame is panned/tilted.

According to the present invention, there is provided a fourth videoapparatus having a pan/tilt determining apparatus which determineswhether an image is in a pan/tilt state for each frame of a movingimage, wherein the pan/tilt determining apparatus includes: means whichcalculates a motion vector, a minimum value of correlative accumulatedvalues, and an average value of the correlative accumulated values foreach of a plurality of motion vector detecting regions set in a videoarea of each frame of a moving image by a typical point matching method;means which specifies a region in which the detected motion vector hashigh reliability of the motion vector detecting regions on the basis ofthe average value of the correlative accumulated values detected foreach of the motion vector detecting regions and which calculates thenumber of detection regions in which the detected motion vectors havehigh reliability; means which determines whether the number of detectionregions in which the detected motion vectors have high reliability isnot less than a first threshold value; means which, when the number ofdetection regions in which the detected moving vectors have highreliability is less than the first threshold value, determines that theframe is not panned/tilted; means which, when the number of detectionregions in which the detected motion vectors have high reliability isnot less than the first threshold value, specifies a detection region inwhich motions of a plurality of types are present and a detection regionin which motions of a plurality of types are not present in thedetection regions in which the motion vectors have high reliability onthe basis of an average value of correlative accumulated values detectedfor the detection regions in which the motion vectors have highreliability and a minimum value of the correlative accumulated values,and which calculates the number of detection regions in which themotions of the plurality of types are present; means which determineswhether the number of detection regions in which the motions of theplurality of types are present is less than a second threshold value;means which, when the number of detection regions in which the motionsof the plurality of types are present is not less than the secondthreshold value, determines that the frame is not panned/tilted; meanswhich, when the number of detection regions in which the motions of theplurality of types are present is less than the second threshold value,specifies detection regions in which motion vectors have similarity inthe detection regions in which the motion vectors have high reliabilityand the motions of the plurality of types are not present on the basisof a motion vector detected for the detection region in which the motionvector has high reliability and the motions of the plurality of typesare not present, and which calculates the number of detection regions inwhich motion vectors have similarity; means which determines whether thenumber of detection regions in which motion vectors have similarity isnot less than a third threshold value; means which, when the number ofdetection regions which are determined as detection regions in which themotion vectors have similarity is less than the third threshold value,determines that the frame is not panned/tilted; and means which, whenthe number of detection regions which are determined as detectionregions in which the motion vectors have similarity is not less than thethird threshold value, determines that the frame is panned/tilted.

According to the present invention, there is provided a third frame rateconverting apparatus which converts a frame rate of a moving image,including: means which defines one of two continuous frames in a movingimage as a current frame, defines the other as a previous frame, anddefines a frame generated between the current frame and the previousframe as a prediction frame, which calculates a shift distance of theprediction image with respect to a previous frame image and a shiftdistance of the prediction image with respect to a current frame image,on the basis of a moving distance of an image between a previous frameimage and a current frame image and a frame rate; means which shifts theprevious frame image by using the shift distance of the prediction imagewith respect to the previous frame image to generate a first virtualprediction image constituted by a first blank portion in which theprevious frame image is not present and a first original image portionin which the previous frame image is present; means which shifts thecurrent frame image by using the shift distance of the prediction imagewith respect to the current frame image to generate a second virtualprediction image constituted by a second blank portion in which thecurrent frame image is not present and a second original image portionin which the current frame image is present; and prediction imagegenerating means which generates the prediction image on the basis ofthe previous frame image, the current frame image, the first virtualprediction image, and the second virtual prediction image, wherein theprediction image generating means, if the first virtual prediction imageand the second virtual prediction image are overlapped, generates aprediction image from the second virtual prediction image in a portionwhere the first blank portion in the first virtual prediction image isoverlapped on the second original image portion in the second virtualprediction image, generates a prediction image from the first virtualprediction image in a portion where the second blank portion in thesecond virtual prediction image is overlapped on the first originalimage portion in the first virtual prediction image, generates aprediction image by an image obtained by weighted-summing the currentframe image and the previous frame image in a portion where the firstblank portion and the second blank portion are overlapped, and generatesa prediction image from one of the first virtual prediction image andthe second virtual prediction image in a portion where the firstoriginal image portion in the first virtual prediction image isoverlapped on the second original image portion in the second virtualprediction image.

According to the present invention, there is provided a fifth videoapparatus having a frame rate converting apparatus which converts aframe rate of a moving image, wherein the frame rate convertingapparatus includes: means which defines one of two continuous frames ina moving image as a current frame, defines the other as a previousframe, and defines a frame generated between the current frame and theprevious frames as a prediction frame, which calculates a shift distanceof the prediction image with respect to a previous frame image and ashift distance of the prediction image with respect to a current frameimage, on the basis of a moving distance of an image between theprevious frame image and a current frame image and a frame rate; meanswhich shifts the previous frame image by using the shift distance of theprediction image with respect to the previous frame image to generate afirst virtual prediction image constituted by a first blank portion inwhich the previous frame image is not present and a first original imageportion in which the previous frame image is present; means which shiftsthe current frame image by using the shift distance of the predictionimage with respect to the current frame image to generate a secondvirtual prediction image constituted by a second blank portion in whichthe current frame image is not present and a second original imageportion in which the current frame image is present; and predictionimage generating means which generates the prediction image on the basisof the previous frame image, the current frame image, the first virtualprediction image, and the second virtual prediction image, wherein theprediction image generating means, if the first virtual prediction imageand the second virtual prediction image are overlapped, generates aprediction image from the second virtual prediction image in a portionwhere the first blank portion in the first virtual prediction image isoverlapped on the second original image portion in the second virtualprediction image, generates a prediction image from the first virtualprediction image in a portion where the second blank portion in thesecond virtual prediction image is overlapped on the first originalimage portion in the first virtual prediction image, generates aprediction image by an image obtained by weighted-summing the currentframe image and the previous frame image in a portion where the firstblank portion and the second blank portion are overlapped, and generatesa prediction image from one of the first virtual prediction image andthe second virtual prediction image in a portion where the firstoriginal image portion in the first virtual prediction image isoverlapped on the second original image portion in the second virtualprediction image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration of a digital camera.

FIG. 2 is a block diagram showing a configuration of an anti-shaking andframe rate converting unit 5 for a luminance signal Y.

FIG. 3 is a timing chart showing timings of writing in and reading fromframe memories A, B, and C.

FIG. 4 is a pattern diagram showing a plurality of motion vectordetecting regions E₁ to E₁₂ set in a video area 100 of each frame.

FIG. 5 is a pattern diagram showing small regions e in the motion vectordetecting regions E₁ to E₁₂ in FIG. 4.

FIG. 6 is a pattern diagram showing a plurality of sampling points S andone typical point R set in the small region e in FIG. 5.

FIG. 7 is a flow chart showing a pan/tilt determining procedureperformed by the pan/tilt determining unit 17.

FIG. 8 is a circuit diagram showing a configuration of a moving distancecontrol unit 18.

FIG. 9 is a pattern diagram showing a previous frame image F₁, a currentframe image F₂, and a prediction image F₁₂.

FIG. 10 is a pattern diagram showing an output obtained by masking amaximum blank range at a predetermined video level in the predictionimage F₁₂.

FIG. 11 shows a basic idea of a method of generating a blank-lessprediction image.

FIG. 12 is a pattern diagram showing a diagram obtained by overlapping aprediction image F_(x), a first virtual prediction image F₁₂, and asecond virtual prediction image F₂₁.

FIG. 13 is a pattern diagram showing types of arrangement patterns ofregions Q1, Q2, R, and T in the prediction image F_(x).

FIG. 14 is a block diagram showing a configuration of an output imagegenerating unit 19.

FIG. 15 is a pattern diagram for explaining an example of a conventionalframe rate converting method.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

An embodiment of the present invention will be described below withreference to the accompanying drawings.

[1] Explanation About Configuration of Digital Camera

FIG. 1 shows a configuration of a digital camera.

A CCD 2 performs photoelectric conversion of an optical image beingincident through a lens 1 to output the optical image as an electricsignal. An output signal (RGB signal) from the CCD 2 is transmitted to aCDS/AGC circuit 3 including a CDS circuit and an AGC circuit. An outputsignal input from the CCD 2 to the CDS/AGC circuit 3 is subjected to acorrelative double sampling process by the CDS circuit and then adjustedin gain by the AGC circuit to have an optimum amplitude. An output fromthe CDS/AGC circuit 3 is converted into a YUV signal by a camera processcircuit 4. Reference symbol Y denotes a luminance signal; U, a colordifference signal representing (B−Y); V, a color difference signalrepresenting (R−Y).

The output signal (YUV signal) from the camera process circuit 4 issubjected to an anti-shaking process by an anti-shaking and frame rateconverting unit 5 and then transmitted to an image compressing circuit6. In the image compressing circuit 6, a compressing process isperformed to the signal (YUV signal) obtained after the anti-shakingprocess. An output signal from the image compressing circuit 6 isrecorded on a recording media 7.

When a compressed image recorded on the recording media 7 is displayedon a television receiver or the like, the compressed image recorded onthe recording media 7 is read and transmitted to an image extendingcircuit 8. In the image extending circuit 8, the compressed image isextended to obtain a YUV signal. The output signal (YUV signal) from theimage extending circuit 8 is subjected to a frame rate convertingprocess by the anti-shaking and frame rate converting unit 5 and thentransmitted to an NTSC encoder 9. In the NTSC encoder 9, the signal (YUVsignal) subjected to the frame rate converting process is NTSC-encodedand output as an external output.

[2] Explanation About Configuration of Anti-Shaking and Frame RateConverting Unit 5

FIG. 2 shows a configuration of the anti-shaking and frame rateconverting unit 5 for the luminance signal Y. Since the anti-shaking andframe rate converting units 5 for the luminance signal Y and the colordifference signals U and V having the same configurations, theconfiguration for the luminance signal Y will be described below.

[2-1] Explanation About Operation in Image Recording

When a photographed image is recorded, the luminance signal Y obtainedby the camera process circuit 4 is written in a first frame memory(frame memory A) 12 through a selector 11. The luminance signal Ywritten in the first frame memory 12 is read at a predetermined timingand written in a second frame memory (frame memory B) 13. The luminancesignal Y written in the second frame memory 13 is read at apredetermined timing and written in a third frame memory (frame memoryC) 14.

The luminance signal Y read from the first frame memory 12 and theluminance signal Y read from the second frame memory 13 are transmittedto a motion vector detecting unit 15 to detect a motion vector or thelike. The motion vector or the like detected by the motion vectordetecting unit 15 is transmitted to a motion correcting unit 16.

The luminance signal Y read from the third frame memory 14 istransmitted to the motion correcting unit 16. In the motion correctingunit 16, on the basis of the motion vector or the like detected by themotion vector detecting unit 15, motion correction is performed to theluminance signal Y read from the third frame memory 14. The luminancesignal Y subjected to the motion correction by the motion correctingunit 16 is transmitted to the image compressing circuit 6.

[2-2] Explanation About Operation in Image Reproduction

An operation performed when a reproduction image is generated from acompressed image recorded on the recording media 7 will be describedbelow.

The compressed luminance signal recorded on the recording media 7 istransmitted to the image extending circuit 8 and extended. The luminancesignal Y obtained by the image extending circuit 8 is transmitted to theanti-shaking and frame rate converting unit 5. A frame rate of theluminance signal Y read from the recording media 7 and obtained by theimage extending circuit 8 is 30 frames/second. In the anti-shaking andframe rate converting unit 5, the luminance signal Y transmitted fromthe image extending circuit 8 at 30 frames/second is converted into aluminance signal Y at 60 frames/second. That is, the frame rate is madedouble.

The luminance signal Y transmitted from the image extending circuit 8 tothe anti-shaking and frame rate converting unit 5 is written in thefirst frame memory (frame memory A) 12 through the selector 11. Theluminance signal Y written in the first frame memory 12 is read at apredetermined timing and written in the second frame memory (framememory B) 13. The luminance signal Y written in the second frame memory13 is read at a predetermined timing and written in the third framememory (frame memory C) 14.

The luminance signal Y read from the first frame memory (frame memory A)12 and the luminance signal Y read from the second frame memory 13 aretransmitted to the motion vector detecting unit 15 to detect a motionvector or the like. The motion vector or the like detected by the motionvector detecting unit 15 is transmitted to a pan/tilt determining unit17. The pan/tilt determining unit 17 determines whether the frame is aframe photographed in a pan or tilt state on the basis of the motionvector or the like detected by the motion vector detecting unit 15.Depending on the determination result, a temporary moving distance MA ofthe frame is calculated.

The temporary moving distance MA calculated by the pan/tilt determiningunit 17 is transmitted to the moving distance control unit 18. Themoving distance control unit 18 calculates a moving distance M used ingeneration of a prediction image on the basis of the temporary movingdistance MA calculated by the pan/tilt determining unit 17 and thedetermination result of the pan/tilt determining unit 17. The movingdistance M calculated by the moving distance control unit 18 istransmitted to an output image generating unit 19.

The luminance signal Y read from the second frame memory 13 and theluminance signal Y read from the third frame memory 14 are transmittedto the output image generating unit 19. The output image generating unit19 generates an output image on the basis of the luminance signal Y readfrom the second frame memory 13, the luminance signal Y read from thethird frame memory 14, and the moving distance M calculated by themoving distance control unit 18. The output image generated by theoutput image generating unit 19 is transmitted to the NTSC encoder 9.

Of the circuits used in frame rate conversion, as the frame memories 12,13, and 14 and the motion vector detecting unit 15, the frame memories12, 13, and 14 and the motion vector detecting unit 15 in the circuitused in an anti-shaking operation can be used. For this reason, a simpleconfiguration can be achieved.

[3] Detailed Explanation About Operation in Image Reproduction

An operation in image reproduction will be described below.

[3-1] Explanation About Timings of Writing in and Reading from FrameMemories A, B, and C

FIG. 3 shows timings of writing in and reading from the frame memoriesA, B, and C. In FIG. 3, a signal EN is an enable signal generated foreach frame of an original video signal, and a signal SEL is a selectionsignal to control a moving distance of the moving distance control unit18. The enable signal EN and the selection signal SEL are given to themoving distance control unit 18.

[3-2] Explanation About Motion Vector Detecting Unit 15

The motion vector detecting unit 15 calculates a motion vector or thelike on the basis of a typical matching method. As shown in FIG. 4, aplurality of motion vector detecting regions E₁ to E₁₂ are set in avideo area 100 of each frame. The motion vector detecting regions E₁ toE₁₂ have equal sizes. Each of the motion vector detecting regions E₁ toE₁₂ is divided into small regions e as shown in FIG. 5. As shown in FIG.6, a plurality of sampling points S and one typical point R are set ineach small region e.

A difference (correlative value at each sampling point S) between avideo signal level (luminance level) of each sampling point S in thesmall region e in a current frame and a video signal level (luminancelevel) of the typical point R in a corresponding small region e in aprevious frame is calculated for each of the motion vector detectingregions E₁ to E₁₂.

Correlative values of sampling points having equal deflections withrespect to the typical point R are accumulatively added between all thesmall regions e in each of the motion vector detecting regions E₁ toE₁₂. Therefore, correlative accumulated values, the number of whichdepends on the number of sampling points S in one small region e arecalculated for each of the motion vector detecting regions E₁ to E₁₂.

In each of the motion vector detecting regions E₁ to E₁₂, a deflectionof a point at which the correlative accumulated value is minimum, i.e.,a deflection having maximum correlativity is extracted as a motionvector V of a corresponding one of the motion vector detecting regionsE₁ to E₁₂.

Furthermore, in the embodiment, a minimum value MIN of the correlativeaccumulated values is calculated for each of the motion vector detectingregions E₁ to E₁₂, and an average value AVE of the correlativeaccumulated values is calculated for each of the motion vector detectingregions E₁ to E₁₂.

[3-3] Explanation About Pan/Tilt Determining Unit 17

FIG. 7 shows a pan/tilt determining procedure performed by the pan/tiltdetermining unit 17 will be described below.

Of the motion vector detection regions, a detection region in which adetected motion vector V has high reliability is specified, and thenumber of detection regions T₁ in which the detected motion vectors Vhave high reliability is calculated (step S1).

In a region having a small change in luminance, a motion vector cannotbe correctly detected. In the region having a small change in luminance,an average value AVE of correlative accumulated values becomes small.Therefore, when the average value AVE of the correlative accumulatedvalues is larger than a predetermined threshold value α, it isdetermined that the motion vector V detected in the motion vectordetection region has high reliability.

It is determined whether the number of detection regions T₁ in which thedetected motion vectors V have high reliability is threshold value θ₁ ormore (step S2). When the number of detection regions T₁ in which thedetected motion vectors V have high reliability is less than θ₁, it isdetermined that the frame is not panned/tilted (step S7), and thetemporary moving distance MA of the frame is set at 0 (step S8).

When the number of detection regions T₁ in which the detected motionvectors V have high reliability is θ₁ or more, of the detection regionsin which the detected motion vectors V have high reliability, adetection region in which motions of a plurality of types are presentand a detection region in which motions of a plurality of types are notpresent are specified, and the number of detection regions T₂ in whichmotions of a plurality of types are present is calculated (step S3).

When the frame is panned or tilted, a motion of one type is present inthe detection region. For this reason, of the detection regions in whichthe detected motion vectors V have high reliability, the number ofdetection regions T₂ in which motions of a plurality of types arepresent is calculated. When a value (AVE/MIN) obtained by dividing theaverage value AVE of the correlative accumulated values by the minimumvalue MIN of the correlative accumulated values is larger than apredetermined threshold value β, it is determined that the motion vectordetection region has motions of a plurality of types.

It is determined whether the number of detection regions T₂ in which themotions of the plurality of types are present is less than a thresholdvalue θ₂ (step S4). When the number of detection regions T₂ in which themotions of the plurality of types are present is the threshold value θ₂or more, it is determined that the frame is not panned/tilted (step S7),and the temporary moving distance MA is set at 0 (step S8).

When the number of detection regions T₂ in which the motions of theplurality of types are present is less than the threshold value θ₂, indetection regions in which detected motion vectors V have highreliability and motions of a plurality of types are not present, adetection region in which motion vectors have similarity is specified,and the number of detection regions T₂ in which motion vectors havesimilarity is calculated (step S5).

It is determined by the following method whether motion vectors havesimilarity. An average value *V of motion vectors in the detectionregions in which the detected motion vector V has high reliability andthe motions of a plurality of types are not present is calculated. Anabsolute value |V−*V| of a difference between the motion vector and theaverage value *V is smaller than a predetermined threshold value γ foreach of the detection regions. When the absolute value is smaller thanthe predetermined threshold value γ, it is determined that the detectedmotion vector in the motion vector detection region has similarity tothe motion vectors detected in these detection regions.

It is determined whether the number of detection regions T₂ in which themotion vectors have similarity is a threshold value θ₃ or more (stepS6). When the number of detection regions T₂ in which the motion vectorshave similarity is less than the threshold value θ₃, it is determinedthat the frame is not panned/tilted (step S7), and the temporary movingdistance MA of the frame is set at 0 (step S8). When the number ofdetection regions T₂ in which the motion vectors have high similarity isthe threshold value θ₃, it is determined that the frame is panned/tilted(step S9), and the temporary moving distance MA of the frame is set atan average value of the motion vectors V in the detection regions (thenumber T₃) in which the motion vectors have similarity and which arespecified in step S5 (step S10).

The processes in steps S5 and S6 may be omitted. More specifically, itis determined in step S4 that the number of detection regions T₂ inwhich the motions of the plurality of types are present is less than thethreshold value θ₂, it may be determined that the frame ispanned/tilted. In this case, the temporary moving distance MA of theframe is set at the average value of the motion vectors in the detectionregions in which the motions of the plurality of types and which arespecified in step S3.

[3-4] Explanation About Moving Distance Control Unit 18

FIG. 8 shows a configuration of the moving distance control unit 18.

The moving distance control unit 18 includes a smoothing circuit 20which smooths the temporary moving distance MA in a direction of timeand a selector 30 in which an output MB from the smoothing circuit 20and “0” are input. A selection signal SEL serving as a control signal isinput to the selector 30.

The selection signal SEL, as shown in FIG. 3, is “1” when differentframes are input to the motion vector detecting unit 15 (the same framesare input to the output image generating unit 19). When the same framesare input to the motion vector detecting unit 15 (different frames areinput to the output image generating unit 19), the selection signal SELis “0”. The selector 30 outputs an input signal “0” as an actual movingdistance M when the selection signal SEL is 1. When the selection signalSEL is 0, the selector 30 outputs the output MB from the smoothingcircuit 20 as the actual moving distance M.

The smoothing circuit 20 includes a multiplier 21, an adder 22, a memory23, and a multiplier 24. An enable signal EN is given to the memory 23as a write/read, control signal.

A moving distance calculated for a first previous frame by the smoothingcircuit 20 is represented by MB_(t-1). The moving distance MB_(t-1)calculated for the first previous frame is stored in the memory 23.

When a moving distance MA_(t) for the current frame is input to thesmoothing circuit 20, the multiplier 21 multiplies the moving distanceMA for the current frame by a coefficient (1−η). On the other hand, themoving distance MB_(t-1) calculated for the first previous frame is readfrom the memory 23 and transmitted to the selector 30 and the multiplier24. The multiplier 24 multiplies the moving distance MB_(t-1) calculatedfor the first previous frame by the coefficient η.

The adder 22 adds a multiplication result η MB_(t-1) of the multiplier24 to a (1−η)·MA_(t) of the multiplier 21. An addition result (ηMB_(t-1)+(1−η)·MA_(t)) of the adder 22 is stored in the memory 23 as amoving distance MB_(t) calculated for the current frame.

More specifically, it is assumed that a moving distance after smoothingbetween the previous frame and a second previous frame is represented byMB_(t-1) and that the moving distance between the current frame and theprevious frame is represented by MA_(t). In this case, a moving distanceMB_(t) obtained after smoothing between the current frame rate and theprevious frame rate is calculated by the following equation (1):MB _(t) =η MB _(t-1)+(1−η)·MA _(t)  (1)where is a coefficient which regulates the degree of smoothing and fallswithin the range of 0≦η≦1. As the coefficient increases, the degree ofsmoothing increases. When a frame determined to be panned/tilted isswitched to a frame determined not to be panned/tilted, a sharpdifference appears on a video image, and an unnatural moving image isobtained. Therefore, as expressed in the following equation (2), thevalues η are switched between the frame determined to be panned/tiltedand the frame determined not to be panned/tilted.frame which is panned/tilted: η=η₁frame which is not panned/tilted: η=η₂(η₁<η₂)  (2)[3-5] Explanation About Output Image Generating Unit 19

The output image generating unit 19 generates an output image on thebasis of two images input from the frame memories 13 and 14 and themoving distance M given by the moving distance control unit 18.

When the two images input from the frame memories 13 and 14 are equal toeach other, the moving distance M is 0. For this reason, one of the twoimages is output as an output image.

When the two images input from the frame memories 13 and 14 aredifferent from each other, on the basis of the images, the movingdistance M given by the moving distance control unit 18, and aconversion magnification m of a preset frame rate, (m−1) predictionimages are generated. The obtained prediction images are output asoutput images. In this case, since the conversion magnification of theframe rate is 2, one prediction image is generated.

Of the two images input from the frame memories 13 and 14, an imagehaving early recording time is called a previous frame, and an imagehaving late recording time is called a current frame.

On the basis of the following equation (3), a shift distance C_(t-1) ofa prediction image for an image of the previous frame is calculated.Note that m=2 is satisfied.C _(t-1) =M/m  (3)

The image of the previous frame is shifted by the shift distance C_(t-1)to generate a prediction image. For example, as shown in FIG. 9, it isassumed that the image of the previous frame is defined as F₁ and thatthe image of the current frame is defined as F₂. In this case, when theimage F₁ of the previous frame is moved by the shift distance C_(t-1) togenerate a prediction image F₁₂.

On the basis of the following equation (4), a shift distance C_(t) ofthe prediction image for an image of the current frame is calculated,and the image of the current frame is moved by the shift distance C_(t),so that a prediction image may be generated.C _(t) =−M/m  (4)

The image of the previous frame may be moved by the shift distanceC_(t-1) to generate a first prediction image, the image of the currentframe may be moved by the shift distance C_(t) to generate a secondprediction image, and the first prediction image and the secondprediction image may be weighted-summed to generate an output image.

When a prediction image is generated by moving an image of an originalframe, a blank portion in which the image of the original frame is notpresent is formed in the prediction image. The maximum range of theportion where the blank is generated is calculated in advance. As shownin FIG. 10, the maximum range S of the blank in the prediction image F₁₂is preferably masked at a predetermined video level (for example, blacklevel) to generate an output image.

When the conversion magnification m of the frame rate is, for example,3, two prediction images are generated. In this case, the shift distanceC_(t-1) of the first prediction image for the image of the previousframe is given by (⅓)×M, and the shift distance C_(t-1) of the secondprediction image for the image of the previous frame is given by (⅔)×M.

As described above, when the prediction image is generated by moving theimage of the original frame, a blank portion where the image of theoriginal frame is not present is formed in the prediction image. Amethod of generating a prediction image which is free from such a blankportion will be described below.

FIG. 11 shows a basic idea of the method of generating a predictionimage.

In FIG. 11, reference symbol F₁ denotes an image of the previous frame,and reference symbol F₂ denotes an image of the current frame.

In this case, it is assumed that a shift distance of a prediction imagefor the image F₁ of the previous frame is expressed by C_(t) and that ashift distance of a prediction image for the image F₂ of the currentframe is expressed by C_(t+1). When the conversion magnification of theframe rate is represented by m, and a moving distance given by themoving distance control unit 18 is represented by M. In this case, thedistances C_(t) and C_(t+1) are calculated by the following equation(5). In this example, m=2 is satisfied.C _(t) =M/mC _(t+1) =−M/m  (5)

The shift distances C_(t) and C_(t+1) consist of horizontal shiftdistances C_(ti), C_((t+1)i) and vertical shift distances C_(tj,)C_((t+1)j), respectively.

A sign of the horizontal shift distance C_(ti) obtained when the imageF₁ of the previous frame is shifted to right is set as a positive (plus)sign, and a sign of the vertical shift distance C_(tj) obtained when theimage F₁ of the previous frame is shift downward is set as a positive(plus) sign.

In the example in FIG. 11, a case in which both the signs of C_(ti) andC_(tj) are positive is explained. A prediction image F₁₂ obtained byshifting the image F₁ of the previous frame by the shift distance C_(t)is generated. The first virtual prediction image F₁₂ is an imageobtained by shifting the image F₁ of the previous frame to right byC_(ti) and downward by C_(tj). A hatched portion S1 in the first virtualprediction image F₁₂ indicates a blank portion where the image F₁ of theoriginal frame is not present.

A second virtual prediction image F₂₁ obtained by shifting the image F₂of the current frame by a shift distance C_(t+1) is generated. Thesecond virtual prediction image F₂₁ is an image obtained by shifting theimage F₂ of the current frame to left by C_(ti) and upward by C_(tj). Ahatched portion S2 in the second virtual prediction image F₂₁ indicatesa blank portion where the image F₂ of the original frame is not present.

On the basis of the image F₁ of the previous frame, the image F₂ of thecurrent frame, the first virtual prediction image F₁₂, and the secondvirtual prediction image F₂₁, a prediction image F_(x) which is freefrom a blank portion is generated.

FIG. 12 is a diagram obtained by overlapping the prediction image F_(x),the first virtual prediction image F₁₂, and the second virtualprediction image F₂₁.

In FIG. 12, a blank portion in the first virtual prediction image F₁₂ isdefined as a region S1, and a blank portion in the second virtualprediction image F₂₁ is defined as a region S2. A portion of the regionS1 which is not overlapped on the region S2 is represented by Q1, and aportion of the region S2 which is not overlapped on the region S1 isrepresented by Q2. Two corner portions where the region S1 and theregion S2 are overlapped are represented by R, and other regions arerepresented by T.

With respect to the regions T in the prediction image F_(x), aprediction image can be generated from any one of the first virtualprediction image F₁₂ and the second virtual prediction image F₂₁.However, in this example, the prediction image is generated from thefirst virtual prediction image F₁₂. With respect to the region Q1 in theprediction image F_(x), an original image is present in the secondvirtual prediction image F₂₁. For this reason, the prediction image isgenerated from the second virtual prediction image F₂₁. With respect tothe region Q2 in the prediction image F_(x), the original image ispresent in the first virtual prediction image F₁₂. For this reason, theprediction image is generated from the first virtual prediction imageF₁₂. With respect to the regions R in the prediction image F_(x), animage obtained by weighted-summing the image F₁ of the previous frameand the image F₂ of the current frame. In this manner, the predictionimage F_(x) which is free from a blank portion can be generated.

The case in which both the signs of the shift distances C_(ti) andC_(tj) are positive is described above. However, depending on acombination of the signs of the shift distances C_(ti) and C_(tj), asarrangement patterns of the regions Q1, Q2, R, and T in the predictionimage F_(x), patterns (P=1 to 4) of four types are present as shown inFIG. 13.

In the patterns P1 to P4, with respect to the region T in the predictionimage F_(x), the prediction image is generated from the first virtualprediction image F₁₂. With respect to the region Q1 in the predictionimage F_(x), the prediction image is generated from the second virtualprediction image F₂₁. With respect to the region Q2 in the predictionimage F_(x), the prediction image is generated from the first virtualprediction image F₁₂. With respect to the region R in the predictionimage F_(x), the prediction image is generated from the image obtainedby weighted-summing the image F₁ of the previous frame and the image F₂of the current frame. In this manner, the prediction image F_(x) whichis free from a blank portion can be generated.

FIG. 14 shows a configuration of the output image generating unit 19 togenerate the prediction image F_(x) described above.

Reference symbol 101 denotes a previous frame shift distance calculatingunit which calculates a shift distance C_(t) of the image F₁ of theprevious frame on the basis of a moving distance M given by the movingdistance control unit 18. Reference numeral 102 denotes a current frameshift distance calculating unit which calculates a shift distanceC_(t+1) of the image F₂ of the current frame on the basis of the movingdistance M given by the moving distance control unit 18.

Reference numeral 103 denotes a previous frame image shifting unit whichgenerates the first virtual prediction image F₁₂ obtained by shiftingthe image F₁ of the previous frame given by the frame memory C (14) bythe shift distance C_(t). Reference numeral 104 denotes a current frameimage shifting unit which generates the second virtual prediction imageF₂₁ obtained by shifting the image F₂ of the current frame given by theframe memory B (13) by the shift distance C_(t+1).

Reference numeral 107 denotes a multiplier which multiplies the image F₁of the previous frame given by the frame memory C (14) by a coefficientδ (0≦δ≦1). Reference numeral 108 denotes a multiplier which multipliesthe image F₂ of the current frame given by the frame memory B (13) by acoefficient (1−δ). Reference numeral 109 denotes an adder which adds amultiplication result of the multiplier 107 and a multiplication resultof the multiplier 108. From the adder 109, an image obtained byweighted-summing the image F₁ of the previous frame and the image F₂ ofthe current frame is output.

The first virtual prediction image F₁₂ output from the previous frameimage shifting unit 103, the second virtual prediction image F₂₁ outputfrom the current frame image shifting unit 104, and the weighted-summedimage output from the adder 109 are input to a selector 110.

Reference numeral 105 denotes a pattern classifying unit whichclassifies patterns into the arrangement patterns of the four typesshown in FIG. 13 on the basis of the signs of the horizontal shiftdistance C_(ti) and the vertical shift distance C_(tj) of the image F1of the previous frame. Reference numeral 106 denotes a selection signaloutput unit which outputs a selection signal SEL to the selector 110 onthe basis of a pattern P classified by the pattern classifying unit 105.

The pattern classifying unit 105 outputs a pattern signal P on the basisof the following conditional equations (6).If C_(ti)≦0 and C_(tj)≦0 then P=1If C_(ti)>0 and C_(tj)≦0 then P=2If C_(ti)≦0 and C_(tj)>0 then P=3If C_(ti)>0 and C_(tj)>0 then P=4  (6)

An operation of the selection signal output unit 106 will be describedbelow. The selection signal output unit 106 outputs a selection signalSEL=1 to a pixel position where the first virtual prediction image F₁₂should be selected, the selection signal output unit 106 outputs aselection signal SEL=2 to a pixel position where the second virtualprediction image F₂₁ should be selected, the selection signal outputunit 106 outputs a selection signal SEL=3 to a pixel position where theweighted-summed image should be selected.

Therefore, in each of the patterns (P=1 to 4) shown in FIG. 13, theselection signal SEL=1 is output to the pixel positions in the regions Tand Q2, the selection signal SEL=2 is output to the pixel position inthe region Q1, and the selection signal SEL=3 is output to the pixelposition in the region R.

More specifically, the selection signal output unit 106 outputsselection signals on the basis of the following conditional equationsdetermined for each pattern where a pixel position is represented by(i,j), the number of pixels in a horizontal direction of one frame isrepresented by I, and the number of pixels in a vertical direction ofone frame is represented by J.

(1) Conditional Equations when P=1 (FIG. 13 (a))if |C _(ti) |≦i<I and |C _(ti) |≦j<J, then SEL=1if 0≦i<(I−C _(ti)) and 0≦j<|C _(tj)| or0≦i<|C _(ti)| and |C _(ti) |≦j<(J−|C _(tj)|), then SEL=2if 0≦i<|C _(ti)| and (J−|C _(tj)|)≦j<J or(I−|C _(ti)|)≦i<I and 0≦j<|C _(tj)|, then SEL=3(2) Conditional Equations when P=2 (FIG. 13 (b))if 0≦i<(I−|C _(ti)|) and |C _(tj) |≦j<J, then SEL=1if |C _(ti) |≦i<I and 0≦j<|C _(tj)| or(I−|C _(ti)|)≦i<I and |C _(tj) |≦j<(J−|C _(tj)|), then SEL=2if 0≦i<|C _(ti)| and 0≦j<|C _(tj)| or(I−|C _(ti)|)≦i<I and (J−|C _(tj)|)≦j<J, then SEL=3(3) Conditional Equations when P=3 (FIG. 13 (c))if |C _(ti) |≦i<I and 0≦j<(J−|C _(ti)|), then SEL=1if 0≦i<|C _(ti)| and |C _(tj) |≦j<J or|C _(ti) |≦i<(I−|C _(ti)|) and (J−|C _(tj)|)≦j<J, then SEL=2if 0≦i<|C _(ti)| and 0≦j<|C _(tj)| or(I−|C _(ti)|)≦i<I and (J−|C _(ti)|)≦j<J, then SEL=3(4) Conditional Equations when P=4 (FIG. 13 (d))if 0≦i<(I−|C _(ti)|) and 0≦j<(J−|C _(tj)|), then SEL=1if |C _(ti) |≦i<(I−|C _(ti)|) and (J−|C _(tj))≦j<J or(I−|C _(ti)|)≦i<I and |C _(tj) |≦j<J, then SEL=2if 0≦i<|C _(ti)| and (J−|C _(tj)|)≦j<J or(I−|C _(ti)|)≦i<I and 0≦j<|C _(tj)|, then SEL=3

When the selector 110 receives the selection signal SEL=1 from theselection signal output unit 106, the selector 110 selects and outputsthe first virtual prediction image F₁₂. When the selector 110 receivesthe selection signal SEL=2 from the selection signal output unit 106,the selector 110 selects and outputs the second virtual prediction imageF₂₁. When the selector 110 receives the selection signal SEL=3 from theselection signal output unit 106, the selector 110 selects and outputs aweighted-summed image. In this manner, a prediction image which is freefrom a blank portion is generated.

1. A frame rate converting apparatus which converts a frame rate of a moving image, comprising: motion detecting means which detects information related to motion of an image for each frame of a moving image to be converted; pan/tilt determining means which determines whether the image is in a pan/tilt state on the basis of the information related to the motion of the image detected by the motion detecting means for each frame of the moving image to be converted; moving distance setting means which sets 0 as a moving distance of an image with respect to a frame the image of which is not determined as an image in a pan/tilt state by the pan/tilt determining means, which calculates a moving distance of an image from the information related to the motion of the image detected by the motion detecting means with respect to a frame the image of which is determined as an image in a pan/tilt state, and which sets the obtained moving distance as a moving distance of the image; and prediction image generating means which generates a prediction image necessary for frame rate conversion on the basis of the moving distance of the image set for each frame by the moving distance setting means.
 2. A frame rate converting apparatus which converts a frame rate of a moving image, comprising: motion detecting means which detects information related to motion of an image for each frame of a moving image to be converted; pan/tilt determining means which determines whether the image is in a pan/tilt state on the basis of the information related to the motion of the image detected by the motion detecting means for each frame of the moving image to be converted; moving distance setting means which sets 0 as a moving distance of an image with respect to a frame the image of which is not determined as an image in a pan/tilt state by the pan/tilt determining means, which calculates a moving distance of an image from the information related to the motion of the image detected by the motion detecting means with respect to a frame the image of which is determined as an image in a pan/tilt state, and which sets the obtained moving distance as a moving distance of the image; time-direction smoothing means which smoothes the moving distance of the image set for each frame by the moving distance setting means in a direction of time; and prediction image generating means which generates a prediction image necessary for frame rate conversion on the basis of the moving distance of the image obtained by the time-direction smoothing means.
 3. The frame rate converting apparatus according to claim 1, wherein the motion detecting means detects a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values as information related to the motion of the image for each of a plurality of motion vector detecting regions set in a video area of each frame by a typical point matching method, and the pan/tilt determining means includes means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability; means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value; means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted; means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present; means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value; means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted; and means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, determines that the frame is panned/tilted.
 4. The frame rate converting apparatus according to claim 2, wherein the motion detecting means detects a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values as information related to the motion of the image for each of a plurality of motion vector detecting regions set in a video area of each frame by a typical point matching method, and the pan/tilt determining means includes means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability; means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value; means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted; means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present; means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value; means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted; and means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, determines that the frame is panned/tilted.
 5. The frame rate converting apparatus according to claim 1, wherein the motion detecting means detects a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values as information related to the motion of the image for each of a plurality of motion vector detecting regions set in a video area of each frame by a typical point matching method, and the pan/tilt determining means includes: means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability; means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value; means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted; means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present; means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value; means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted; means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, specifies detection regions in which motion vectors have similarity in the detection regions in which the motion vectors have high reliability and the motions of the plurality of types are not present on the basis of a motion vector detected for the detection region in which the motion vector has high reliability and the motions of the plurality of types are not present, and which calculates the number of detection regions in which motion vectors have similarity; means which determines whether the number of detection regions in which motion vectors have similarity is not less than a third threshold value; means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is less than the third threshold value, determines that the frame is not panned/tilted; and means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is not less than the third threshold value, determines that the frame is panned/tilted.
 6. The frame rate converting apparatus according to claim 2, wherein the motion detecting means detects a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values as information related to the motion of the image for each of a plurality of motion vector detecting regions set in a video area of each frame by a typical point matching method, and the pan/tilt determining means includes: means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability; means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value; means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted; means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present; means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value; means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted; means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, specifies detection regions in which motion vectors have similarity in the detection regions in which the motion vectors have high reliability and the motions of the plurality of types are not present on the basis of a motion vector detected for the detection region in which the motion vector has high reliability and the motions of the plurality of types are not present, and which calculates the number of detection regions in which motion vectors have similarity; means which determines whether the number of detection regions in which motion vectors have similarity is not less than a third threshold value; means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is less than the third threshold value, determines that the frame is not panned/tilted; and means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is not less than the third threshold value, determines that the frame is panned/tilted.
 7. A video apparatus having a frame rate converting apparatus which converts a frame rate of a moving image, wherein the frame rate converting apparatus includes: motion detecting means which detects information related to motion of an image for each frame of a moving image to be converted; pan/tilt determining means which determines whether the image is in a pan/tilt state on the basis of the information related to the motion of the image detected by the motion detecting means for each frame of the moving image to be converted; moving distance setting means which sets 0 as a moving distance of an image with respect to a frame the image of which is not determined as an image in a pan/tilt state by the pan/tilt determining means, which calculates a moving distance of an image from the information related to the motion of the image detected by the motion detecting means with respect to a frame the image of which is determined as an image in a pan/tilt state, and which sets the obtained moving distance as a moving distance of the image; and prediction image generating means which generates a prediction image necessary for frame rate conversion on the basis of the moving distance of the image set for each frame by the moving distance setting means.
 8. A video apparatus having a frame rate converting apparatus which converts a frame rate of a moving image, wherein the frame rate converting apparatus includes: motion detecting means which detects information related to motion of an image for each frame of a moving image to be converted; pan/tilt determining means which determines whether the image is in a pan/tilt state on the basis of the information related to the motion of the image detected by the motion detecting means for each frame of the moving image to be converted; moving distance setting means which sets 0 as a moving distance of an image with respect to a frame the image of which is not determined as an image in a pan/tilt state by the pan/tilt determining means, which calculates a moving distance of an image from the information related to the motion of the image detected by the motion detecting means with respect to a frame the image of which is determined as an image in a pan/tilt state, and which sets the obtained moving distance as a moving distance of the image; time-direction smoothing means which smooths the moving distance of the image set for each frame by the moving distance setting means in a direction of time; and prediction image generating means which generates a prediction image necessary for frame rate conversion on the basis of the moving distance of the image obtained by the time-direction smoothing means.
 9. A pan/tilt determining apparatus which determines whether an image is in a pan/tilt state for each frame of a moving image, comprising: means which calculates a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values for each of a plurality of motion vector detecting regions set in a video area of each frame of a moving image by a typical point matching method; means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability; means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value; means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted; means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present; means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value; means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted; and means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, determines that the frame is panned/tilted.
 10. A pan/tilt determining apparatus which determines whether an image is in a pan/tilt state for each frame of a moving image, comprising: means which calculates a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values for each of a plurality of motion vector detecting regions set in a video area of each frame of a moving image by a typical point matching method; means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability; means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value; means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted; means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present; means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value; means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted; means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, specifies detection regions in which motion vectors have similarity in the detection regions in which the motion vectors have high reliability and the motions of the plurality of types are not present on the basis of a motion vector detected for the detection region in which the motion vector has high reliability and the motions of the plurality of types are not present, and which calculates the number of detection regions in which motion vectors have similarity; means which determines whether the number of detection regions in which motion vectors have similarity is not less than a third threshold value; means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is less than the third threshold value, determines that the frame is not panned/tilted; and means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is not less than the third threshold value, determines that the frame is panned/tilted.
 11. A video apparatus having a pan/tilt determining apparatus which determines whether an image is in a pan/tilt state for each frame of a moving image, wherein the pan/tilt determining apparatus includes: means which calculates a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values for each of a plurality of motion vector detecting regions set in a video area of each frame of a moving image by a typical point matching method; means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability; means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value; means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted; means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present; means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value; means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted; and means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, determines that the frame is panned/tilted.
 12. A video apparatus having a pan/tilt determining apparatus which determines whether an image is in a pan/tilt state for each frame of a moving image, wherein the pan/tilt determining apparatus includes: means which calculates a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values for each of a plurality of motion vector detecting regions set in a video area of each frame of a moving image by a typical point matching method; means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability; means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value; means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted; means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present; means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value; means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted; means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, specifies detection regions in which motion vectors have similarity in the detection regions in which the motion vectors have high reliability and the motions of the plurality of types are not present on the basis of a motion vector detected for the detection region in which the motion vector has high reliability and the motions of the plurality of types are not present, and which calculates the number of detection regions in which motion vectors have similarity; means which determines whether the number of detection regions in which motion vectors have similarity is not less than a third threshold value; means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is less than the third threshold value, determines that the frame is not panned/tilted; and means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is not less than the third threshold value, determines that the frame is panned/tilted.
 13. A frame rate converting apparatus which converts a frame rate of a moving image, comprising: means which defines one of two continuous frames in a moving image as a current frame, defines the other as a previous frame, and defines a frame generated between the current frame and the previous frame as a prediction frame, which calculates a shift distance of the prediction image with respect to a previous frame image and a shift distance of the prediction image with respect to a current frame image, on the basis of a moving distance of an image between a previous frame image and a current frame image and a frame rate; means which shifts the previous frame image by using the shift distance of the prediction image with respect to the previous frame image to generate a first virtual prediction image constituted by a first blank portion in which the previous frame image is not present and a first original image portion in which the previous frame image is present; means which shifts the current frame image by using the shift distance of the prediction image with respect to the current frame image to generate a second virtual prediction image constituted by a second blank portion in which the current frame image is not present and a second original image portion in which the current frame image is present; and prediction image generating means which generates the prediction image on the basis of the previous frame image, the current frame image, the first virtual prediction image, and the second virtual prediction image, wherein the prediction image generating means, if the first virtual prediction image and the second virtual prediction image are overlapped, generates a prediction image from the second virtual prediction image in a portion where the first blank portion in the first virtual prediction image is overlapped on the second original image portion in the second virtual prediction image, generates a prediction image from the first virtual prediction image in a portion where the second blank portion in the second virtual prediction image is overlapped on the first original image portion in the first virtual prediction image, generates a prediction image by an image obtained by weighted-summing the current frame image and the previous frame image in a portion where the first blank portion and the second blank portion are overlapped, and generates a prediction image from one of the first virtual prediction image and the second virtual prediction image in a portion where the first original image portion in the first virtual prediction image is overlapped on the second original image portion in the second virtual prediction image.
 14. A video apparatus having a frame rate converting apparatus which converts a frame rate of a moving image, wherein the frame rate converting apparatus includes: means which defines one of two continuous frames in a moving image as a current frame, defines the other as a previous frame, and defines a frame generated between the current frame and the previous frames as a prediction frame, which calculates a shift distance of the prediction image with respect to a previous frame image and a shift distance of the prediction image with respect to a current frame image, on the basis of a moving distance of an image between a previous frame image and a current frame image and a frame rate; means which shifts the previous frame image by using the shift distance of the prediction image with respect to the previous frame image to generate a first virtual prediction image constituted by a first blank portion in which the previous frame image is not present and a first original image portion in which the previous frame image is present; means which shifts the current frame image by using the shift distance of the prediction image with respect to the current frame image to generate a second virtual prediction image constituted by a second blank portion in which the current frame image is not present and a second original image portion in which the current frame image is present; and prediction image generating means which generates the prediction image on the basis of the previous frame image, the current frame image, the first virtual prediction image, and the second virtual prediction image, wherein the prediction image generating means, if the first virtual prediction image and the second virtual prediction image are overlapped, generates a prediction image from the second virtual prediction image in a portion where the first blank portion in the first virtual prediction image is overlapped on the second original image portion in the second virtual prediction image, generates a prediction image from the first virtual prediction image in a portion where the second blank portion in the second virtual prediction image is overlapped on the first original image portion in the first virtual prediction image, generates a prediction image by an image obtained by weighted-summing the current frame image and the previous frame image in a portion where the first blank portion and the second blank portion are overlapped, and generates a prediction image from one of the first virtual prediction image and the second virtual prediction image in a portion where the first original image portion in the first virtual prediction image is overlapped on the second original image portion in the second virtual prediction image. 